{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Online store advertising\n",
    "\n",
    "## 1. Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-07-25T15:29:10.289133Z",
     "start_time": "2023-07-25T15:29:10.241132200Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "DF_ads = pd.read_csv(\"./dataset/3_2_advertising.csv\")\n",
    "# DF_ads\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Analyze to data\n",
    "\n",
    "use methods of data visualiaztion to deduce the most relation feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-07-25T15:29:49.975344800Z",
     "start_time": "2023-07-25T15:29:49.902183600Z"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns  # A data-visualization tool of Seaborn of Statistics\n",
    "\n",
    "\"draw heatmap(热力图) to all label and features\"\n",
    "sns.heatmap(DF_ads.corr(), cmap=sns.cm._flare_lut, annot=True)\n",
    "plt.show()\n",
    "\n",
    "\"use scatter to dispaly relation of 2(ads method and sale)\"\n",
    "sns.pairplot(DF_ads,\n",
    "             x_vars=['wechat', 'weibo', 'others'],\n",
    "             y_vars='sales',\n",
    "             height=4, aspect=1, kind='scatter')\n",
    "plt.show()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Clean and format of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"as we know the 'wecaht' are most corelate feature to the label from upon seies\"\n",
    "X = np.array(DF_ads.wechat)\n",
    "y = np.array(DF_ads.sales)\n",
    "print(\"rank of X: \", X.ndim)\n",
    "print(\"shape of X: \", X.shape)\n",
    "# print(X)\n",
    "\n",
    "\"format to 2D tensor, so that ml model can received\"\n",
    "X = X.reshape((len(X), 1))\n",
    "y = y.reshape((len(y), 1))\n",
    "print(\"rank of X: \", X.ndim)\n",
    "print(\"shape of X: \", X.shape)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have 2 dimension, 1 for sample(样本) (just meaning of 1 row), 1 for feature(特征) (the data or this row)\n",
    "\n",
    "Then we need to do is:\n",
    "  1. split dataset into train-set and test-set   \n",
    "\n",
    "  2. unifrom(归一化) to dataset like below function\n",
    "   \n",
    "      $$x^{'}=\\frac{x-min(x)}{max(X)-min(X)}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "print(X_train.shape, '---', X_train.ndim)\n",
    "print(y_train.shape, '---', y_train.ndim)\n",
    "\n",
    "\n",
    "# def scaler(train :np.array, test: np.array): # data after split did not compatiable with array, maybe more than some dims\n",
    "def scaler(train, test):\n",
    "    \"Use ourselves' unifrom method instead of MaxScaler of sklearn.preprocessing\"\n",
    "    # max and min value of train-set\n",
    "    min = train.min(axis=0)\n",
    "    max = train.max(axis=0)\n",
    "    # the diff (区间值)\n",
    "    gap = max - min\n",
    "\n",
    "    \"\"\"\n",
    "    All data substruct by min value, and then divide by diff of range(区间)\n",
    "    So it turn into 0~1 maybe more or less by test data\n",
    "    \"\"\"\n",
    "    train -= min\n",
    "    train /= gap\n",
    "    test -= min\n",
    "    test /= gap\n",
    "\n",
    "    \"return datas that have been compressed\"\n",
    "    return train, test\n",
    "\n",
    "\n",
    "# function also equivalent to this, just equvalation above\n",
    "# x_norm = (x_data -np.min(x_data)) / np(max(x_data)-np.min(x_data)).values\n",
    "\n",
    "# NOTICE: cannot use diff(gap) by the max-min from `testset`!!! so will use the train-set's diff\n",
    "X_train, X_test = scaler(X_train, X_test)\n",
    "y_train, y_test = scaler(y_train, y_test)\n",
    "\n",
    "\n",
    "\"Display datas that have been precessor, to see the difference on above notes\"\n",
    "plt.plot(X_train, y_train, 'r.', label=\"Trainning data\")\n",
    "plt.xlabel(\"wechat\")\n",
    "plt.ylabel('sales')\n",
    "plt.legend()  # display 图例-legend\n",
    "plt.show()\n",
    "# notice the `r` and `r.`\n",
    "plt.plot(X_train, y_train, 'r', label=\"Trainning data\")\n",
    "plt.show()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Choose model\n",
    "\n",
    "- It's 2 array of dataset, feature and label. (2 Nx1 tensor).\n",
    "- So it's simple to find the `linear regression` as the model. (from the 2nd view of data above)\n",
    "\n",
    "### 4.1 predict what the function is -- h(x)\n",
    "\n",
    "$$ y' = wx +b $$\n",
    "also:\n",
    "$$ h(x) = wx + b $$\n",
    "\n",
    "### 4.2 loss and loss-function\n",
    "\n",
    "To deduce which is better from 2 models between `3x+5` and `100x+1` somehow...\n",
    "\n",
    "1. If model prefic absolutely precise, it will 0 (of cost/loss).\n",
    "2. If did not match, will bring some loss value.\n",
    "\n",
    "The `loss funtion` $L (w, b)$ which will use to caculate the average loss.\n",
    "\n",
    "Will be call as `cost function` (代价/成本函数) somewhere, and noted it as $J(\\theta)$, And the $\\theta$ is just `w and b`， $$J(\\theta) = L(w,b)$$\n",
    "\n",
    "> NOTICE: loss function is about `w` and `b`, but not about `x` and `y`, we should turn our minds about it. Why:\n",
    "\n",
    "- As in the trainning, the x and y are fixed.\n",
    "- So w and b turn into variable in loss-func.\n",
    "\n",
    "#### 4.2.1 Loss function that used in `Regression`\n",
    "\n",
    "- Mean Square Error, MSE (均方误差), called squre-loss (平方损失) or L2-loss function also.\n",
    "- Mean Abolute Error, MAE, called L1-loss also.\n",
    "\n",
    "#### 4.2.2 Loss function that used in `Classification`\n",
    "\n",
    "- Corss-entropy loos (交叉熵损失) function, so called 均方差， 最小二乘法\n",
    "- Hinge loss (多分类SVM损失)  function\n",
    "\n",
    "\n",
    "### 4.2.3 Detail\n",
    "\n",
    "1. For every  sample: Its *predict* and *label*'s diff is $(y-y')$, and $y'=wx+b$, so `loss value` is related with w and b.\n",
    "\n",
    "2. If we promte the $(y-y')$, squre of it....(all are positve, non-negative), so called `square loss` of every sample:   $(y-y')^2$\n",
    "\n",
    "3. Then, we need sumup are `square-loss` of all sample:\n",
    "$$\n",
    "    ( y（x(1)-y'(x(1) )^2 + \n",
    "    ( y（x(2)-y'(x(2) )^2 + \n",
    "    ... +\n",
    "    ( y（x(n)-y'(x(n) )^2  \n",
    "$$\n",
    "And as sumup format is:\n",
    "\n",
    "\n",
    "$$\n",
    "\\sum\\limits_{(x,y)\\in{D}} (y-h(x))^2\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def MSE_loss(X, y, weight, bias):\n",
    "    \"\"\"Ourselves' MSE loss function\"\"\"\n",
    "    y_hat = weight * X + bias\n",
    "    loss = y_hat - y  # every diff from prefit and label\n",
    "    squre_loss_sum = np.sum(loss**2)\n",
    "    # print(squre_loss_sum)\n",
    "    # print(2*len(X))\n",
    "    # cost = squre_loss_sum / 2*len(X)  # not equal when a/b * c and a/ (b*c): 5/5 *2 = 2, 5/(5*2) =1/2\n",
    "    cost = squre_loss_sum / (2*len(X))  # average for divide  2n\n",
    "    return cost\n",
    "\n",
    "\n",
    "weight = 5\n",
    "bias = 3\n",
    "print(\n",
    "    f\"When `weight`=={weight}, `bais`=={bias}, the loss={ MSE_loss(X_train, y_train, weight, bias)}\")\n",
    "weight = 100\n",
    "bias = 1\n",
    "print(\n",
    "    f\"When `weight`=={weight}, `bais`=={bias}, the loss={ MSE_loss(X_train, y_train, weight, bias)}\")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Use `gradient descent` (梯度下降法)  to find best parameters\n",
    "\n",
    "We have noticed that (5,3) is better than (100,1) of (w, b). \n",
    "\n",
    "But how to let bare machine to know which is best?\n",
    "1. Every guess (parameters) should be better than previous one.\n",
    "2. So the convex function(凸函数) and global lower point (全局最节点) were here.\n",
    "\n",
    "The original *Mean Squre Error* function:\n",
    "$$\n",
    "MSE = L(w,b) = \\frac{1}{2N} \\sum\\limits_{(x,y)\\in{d}} (y-(wx+b))^2\n",
    "$$\n",
    "\n",
    "The `x` and `y` could be considered as constant fixed value... So `L` will only changed by the `w` and `b`.\n",
    "\n",
    "And the function is a `quadratic function` (二次函数), we could just think it was a parabola with upward opening (开口向上的抛物线)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def fx(x): return 2*x**2 + 1\n",
    "\n",
    "\n",
    "x = np.linspace(-5, 5, 100)\n",
    "Y = fx(x)\n",
    "plt.plot(x, Y, 'r', \"f(x) ->2d\")\n",
    "plt.show()\n",
    "\n",
    "\n",
    "def hx(x, y): return 2*x**2 + y**2 + 3\n",
    "\n",
    "\n",
    "y = np.linspace(-5, 5, 100)\n",
    "X, Y = np.meshgrid(x, y)\n",
    "Z = hx(X, Y)\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.plot_surface(X, Y, Z, cmap='viridis')\n",
    "plt.show()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 The implementation of gradient descent\n",
    "\n",
    "Just change the `w` and `b` to let the `L` (loss) more and more close to the lowest point.\n",
    "\n",
    "Just a  dynmaic plan: go the lower loss point every step.\n",
    "\n",
    "> Caculate the derivative (导数) !\n",
    "\n",
    "The `gradient` means the vector of loss (current and previous), will `equal to` the `derivative` ?\n",
    "\n",
    "- `Gradient ==  derivative` when there is only one `weight` in equation\n",
    "- More than one weight the gradient will be `partial derivative` (偏导数)\n",
    "\n",
    "So we get the gradient by derivation with loss curve/functon. And the `gradient` have serval properties:\n",
    "\n",
    "1. Direction (the positive/negative of gradient)\n",
    "2. Len/Size/Norm (the magnitude of the tangential tilt (切线倾斜的幅度) )\n",
    "\n",
    "\n",
    "> So how to calculate the gradient in math:\n",
    "\n",
    "~~Notice that will be `x times` but not `square-loss` in partial derviation~~\n",
    "\n",
    "It's the derviation of `function of funtion` (复合函数)\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\n",
    "Gradient &= \\frac{\\partial}{\\partial W} L(W) \\\\\n",
    "         &= \\frac{\\partial}{\\partial W}  \\frac{1}{2N} \\sum \\limits_{(x,y)\\in D}  \n",
    "            (y- h(x)^2)  \\\\\n",
    "         &= \\frac{\\partial}{\\partial W}  \\frac{1}{2N} \\sum \\limits_{(x,y)\\in D} \n",
    "            (y- (w*x)) * X \n",
    "\n",
    "\\end {align}\n",
    "$$\n",
    "\n",
    "Could also be write as:\n",
    "$$\n",
    "\n",
    "Gradient = \\frac{\\partial}{\\partial W}  \\frac{1}{2N} \\sum \\limits_{i=1}^{N} \n",
    "            (y^{(i)}- (w*x^{(i)})) * X^{(i)}\n",
    "$$\n",
    "\n",
    "   ```python\n",
    "   \"A simple implementaion of gradient evaluator\"\n",
    "      y_hat = weight * X + bias  # assumed function model\n",
    "\n",
    "      loss = y_hat - y  # get the loss, [a,b,c....] [dx]\n",
    "\n",
    "   \"X.T is (1,N) and dot the loss of (N,) will just be the dy/dx\"\n",
    "      # derivation of weight, len(X) are count of sample\n",
    "      derivative_weight = X.T.dot(loss) / len(X)\n",
    "\n",
    "      # derivation of bias, len(X) are count of sample\n",
    "      derivative_bias = sum(loss) * 1 / len(X)\n",
    "   ```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 The learning rate (学习速率)\n",
    "\n",
    "$\\alpha$  as a weight of reduce formula(递推公式):\n",
    "\n",
    "$$\n",
    "W = W - \\alpha * \\frac{\\partial}{\\partial{W}} L(W)\n",
    "$$\n",
    "\n",
    "ae:\n",
    "\n",
    "$$\n",
    "W = W - \\frac {\\alpha} {N}  *\n",
    "        \\sum \\limits_{i=1}^{N}\n",
    "        (Y^{(i)} - (W * x^{(i)} )) * x^{(i)}\n",
    "$$\n",
    "\n",
    "\n",
    "code:\n",
    "```py\n",
    "weight = weight - alpha * derivative_weight # update the weight by alpha(learning reate)\n",
    "bias = bias - alpha * derivative_bias\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\"A gradient decent method\"\n",
    "\n",
    "\n",
    "def gradient_descent(X, y, w, b, lr, iter):\n",
    "    \"\"\"\n",
    "    iter: iteration times\n",
    "    lr: learning rate\n",
    "    \"\"\"\n",
    "    l_history = np.zeros(iter)  # recorded the (l)oss array\n",
    "    w_hishory = np.zeros(iter)  # recored the  (w)weight\n",
    "    b_history = np.zeros(iter)  # recored the (b)ias\n",
    "\n",
    "    for i in range(iter):\n",
    "        y_hat = w*X + b\n",
    "        loss = y_hat - y\n",
    "\n",
    "        \"cacualte the partial derivative of weight, bias\"\n",
    "        derivative_w = X.T.dot(loss) / len(X)\n",
    "        derivative_b = sum(loss) * 1 / len(X)\n",
    "\n",
    "        \"use learning rate to implement gradient descent\"\n",
    "        w = w - lr * derivative_w\n",
    "        b = b - lr * derivative_b\n",
    "\n",
    "        \"recored the loss and mapping weight and bias\"\n",
    "        l_history[i] = MSE_loss(X, y, w, b)\n",
    "        w_hishory[i] = w\n",
    "        b_history[i] = b\n",
    "\n",
    "    return l_history, w_hishory, b_history\n",
    "\n",
    "\n",
    "\"Give it a  TRY!\"\n",
    "iterations = 100\n",
    "alpha = 1  # initial learning_rate =1\n",
    "weight = -5\n",
    "bias = 3\n",
    "\n",
    "print(\"Current loss:\", MSE_loss(X_train, y_train, weight, bias))\n",
    "\n",
    "# plot current function model\n",
    "plt.plot(X_train, y_train, 'r.', label=\"Training data\")\n",
    "\n",
    "# plot current 假设函数\n",
    "line_X = np. linspace(X_train.min(), X_train.max(), 500)  # X domain\n",
    "\"we predict the function is `w*x+b``\"\n",
    "line_y = [weight*xx + bias for xx in line_X]\n",
    "plt.plot(line_X, line_y, 'b--', label='current hypothesis')\n",
    "plt.xlabel('wechat')\n",
    "plt.ylabel('sales')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Noticed that the model is differecnet!!!\n",
    "\n",
    "Because the initial `weight` and `bias` we choosed is redicous. Needing the gradient by the `learning rate` adjust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def plot():\n",
    "    loss_history, weight_history, bias_history = gradient_descent(\n",
    "        X_train, y_train, weight, bias, alpha, iterations)\n",
    "\n",
    "    # # \"Plot the current model\"\n",
    "    # \"Plot the loss cruve\"\n",
    "    # plt.plot(loss_history, 'g--', label='loss cruve')\n",
    "    # plt.xlabel('Iterations')\n",
    "    # plt.ylabel('Loss')\n",
    "    # plt.legend()\n",
    "    # plt.show()\n",
    "\n",
    "    # # 绘制 last 的函数模型\n",
    "    # plt.plot(X_train, y_train, 'r.', label='Training data')  # 显示训练数据\n",
    "    # line_X = np.linspace(X_train.min(), X_train.max(), 500)  # X值域\n",
    "    # line_y = [weight_history[-1]*xx + bias_history[-1]\n",
    "    #           for xx in line_X]  # 假设函数\n",
    "    # plt.plot(line_X, line_y, 'b--', label='Current hypothesis')  # 显示当前假设函数\n",
    "    # plt.xlabel('wechat')\n",
    "    # plt.ylabel('slaes')\n",
    "    # plt.legend()\n",
    "    # plt.show()\n",
    "\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 4))\n",
    "\n",
    "    ax1.plot(loss_history, 'g--', label='loss cruve')\n",
    "    ax1.set_xlabel('Iterations')\n",
    "    ax1.set_ylabel('Loss')\n",
    "    ax1.set_title('Loss Curve')\n",
    "\n",
    "    ax2.plot(X_train, y_train, 'r.', label='Training data')\n",
    "    line_X = np.linspace(X_train.min(), X_train.max(), 500)  # X value range\n",
    "    line_y = [weight_history[-1]*xx + bias_history[-1]\n",
    "              for xx in line_X]  # Hypothesis function\n",
    "    ax2.plot(line_X, line_y, 'b--', label='Current hypothesis')\n",
    "    ax2.set_xlabel('WeChat')\n",
    "    ax2.set_ylabel('Sales')\n",
    "    ax2.legend()\n",
    "\n",
    "    plt.subplots_adjust(wspace=0.5)\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "iterations = 100\n",
    "alpha = 1  # initial learning_rate =1\n",
    "weight = -5\n",
    "bias = 3\n",
    "\n",
    "plot()\n",
    "\n",
    "# alpha = 0.01\n",
    "# plot()\n",
    "\n",
    "alpha = 0.5\n",
    "plot()\n",
    "\n",
    "# alpha = 0.8\n",
    "# plot()\n",
    "\n",
    "\n",
    "\"In differecnt iterations\"\n",
    "fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))\n",
    "for ax, iter in [(ax1, 20), (ax2, 50), (ax3, 100)]:\n",
    "    loss_history, weight_history, bias_history = gradient_descent(\n",
    "        X_train, y_train, weight, bias, alpha, iter)\n",
    "\n",
    "    ax.set_xlabel('intertions')\n",
    "    ax.set_ylabel('loss')\n",
    "    ax.plot(loss_history, 'b--', label='current  ')\n",
    "plt.title(\"Loss decrement in different iterations by 0.5 of learning_rate\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "learning_rate = 0.5\n",
    "iteration = 50\n",
    "\n",
    "\n",
    "loss_history, weight_history, bias_history = gradient_descent(\n",
    "    X_train, y_train, weight, bias, learning_rate, iteration)\n",
    "print(\"Current loss from train: \", loss_history[-1])\n",
    "print(\"The test set loss: \", MSE_loss(\n",
    "    X_test, y_test, weight_history[-1], bias_history[-1]))\n",
    "\n",
    "plt.plot(loss_history, 'r.', label=\"train loss cruve\")\n",
    "\n",
    "loss_history, weight_history, bias_history = gradient_descent(\n",
    "    X_test, y_test, weight, bias, learning_rate, iteration)\n",
    "\n",
    "plt.plot(loss_history, 'b--',  label=\"test loss cruve\")\n",
    "\n",
    "plt.xlabel(\"interations\")\n",
    "plt.ylabel(\"loss\")\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. The multi-dimensions funciton (multiple featrues)\n",
    "\n",
    "we think the `bias` * `dummy`, just the $W_0 * X_0$ as a line of `bias`:\n",
    "- $W_0$ ==> bias\n",
    "- $X_0$ ==> dummy\n",
    "\n",
    "So the new regression funciton will be:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "y' &=h(x) = w^Tx \\\\\n",
    "    &= W_0 X_0 +W_1 X_1 + W2_X2 + ....\n",
    "\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "X = np.array(DF_ads)  # contain all features\n",
    "X = np.delete(X, [3], axis=1)  # delete the label (row 3)\n",
    "y = np.array(DF_ads.sales)\n",
    "print(\"tensor X's dimension/rank: \", X.ndim)\n",
    "print(\"tensor X's shape: \", X.shape)\n",
    "# print(X)\n",
    "\n",
    "print(y.shape)\n",
    "\"Turn the y into tensor, and the -1 is meaning of len(y)\"\n",
    "y = y.reshape(-1, 1)\n",
    "print(y.shape)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have sampe (200,1), and the feature (200,3)\n",
    "\n",
    "And then the is **caculation** of `W` and `X`, like $[w_0,w_1,w_2,w_3]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "# for i in range(3):\n",
    "#     y_hat = y_hat+ weight[i]*X[i]\n",
    "#     y_hat = yaht+bias\n",
    "\n",
    "\n",
    "def scaler(train, test):\n",
    "    \"Use ourselves' unifrom method instead of MaxScaler of sklearn.preprocessing\"\n",
    "    # max and min value of train-set\n",
    "    min = train.min(axis=0)\n",
    "    max = train.max(axis=0)\n",
    "    # the diff (区间值)\n",
    "    gap = max - min\n",
    "\n",
    "    \"\"\"\n",
    "    All data substruct by min value, and then divide by diff of range(区间)\n",
    "    So it turn into 0~1 maybe more or less by test data\n",
    "    \"\"\"\n",
    "    train -= min\n",
    "    train /= gap\n",
    "    test -= min\n",
    "    test /= gap\n",
    "\n",
    "    \"return datas that have been compressed\"\n",
    "    return train, test\n",
    "\n",
    "\n",
    "X_train, X_test = scaler(X_train, X_test)\n",
    "y_train, y_test = scaler(y_train, y_test)\n",
    "\n",
    "\n",
    "def dummpy(X):\n",
    "    \"To avoid the loop above\"\n",
    "    x0_train = np.ones((len(X), 1))\n",
    "    # print(x0_train.shape)\n",
    "    X = np.append(x0_train, X, axis=1)\n",
    "    # print(X_train.shape)\n",
    "    # print(X_train) # one col of 1 in front\n",
    "    return X\n",
    "\n",
    "\n",
    "X_train = dummpy(X_train)\n",
    "X_test = dummpy(X_test)\n",
    "\n",
    "\n",
    "# multiple features ver. `loss function`` (MSE)\n",
    "def loss_function(X, y, W):\n",
    "    y_hat = X.dot(W.T)  # Xi * Wi\n",
    "    diff = y_hat.reshape(len(y_hat), 1) - y\n",
    "    cost = np.sum(diff**2) / (2*len(X))  # the MSE\n",
    "    return cost\n",
    "\n",
    "\n",
    "# multiple features ver. `gradient descent`\n",
    "def gradient_descent(X, y, W, lr, iter):\n",
    "    l_history = np.zeros(iter)\n",
    "    w_hishory = np.zeros((iter, len(W)))\n",
    "\n",
    "    for i in range(iter):\n",
    "        # t tensor and caculate the diff/loss\n",
    "        y_hat = X.dot(W)\n",
    "        loss = y_hat.reshape(len(y_hat), 1) - y\n",
    "\n",
    "        \"cacualte the partial derivative of weight, bias\"\n",
    "        derivative_W = X.T.dot(loss) / (2*len(X))\n",
    "        derivative_W = derivative_W.reshape(len(derivative_W))\n",
    "\n",
    "        \"use learning rate to implement gradient descent\"\n",
    "        W = W - lr * derivative_W\n",
    "\n",
    "        \"recored the loss and mapping weight\"\n",
    "        l_history[i] = loss_function(X, y, W)\n",
    "        w_hishory[i] = W\n",
    "\n",
    "    return l_history, w_hishory\n",
    "\n",
    "\n",
    "def linear_regression(X, y, weight, alpha, iterations):\n",
    "    loss_history, weight_history = gradient_descent(\n",
    "        X, y, weight, alpha, iterations)\n",
    "\n",
    "    print(\"------The finnal loss: \", loss_history[-1])\n",
    "    y_pred = X.dot(weight_history[-1])  # use last weight\n",
    "    training_acc = 100 - np.mean(np.abs(y_pred-y))*100\n",
    "    print(\"------linear_regression trainning accuracy rate: \", training_acc)\n",
    "\n",
    "    return loss_history, weight_history\n",
    "\n",
    "\n",
    "iterations = 800\n",
    "alpha = 0.03\n",
    "# the weight vector, weight[0] will be the original bias, other will be dummpy\n",
    "weight = np.array([2.7, 1, 1, 1])\n",
    "print(\"Current loos is: \", loss_function(X_train, y_train, weight))\n",
    "\n",
    "# tranning\n",
    "loss_history, weight_history = linear_regression(\n",
    "    X_train, y_train, weight, alpha, iterations)\n",
    "\n",
    "print(\"Weight history: \", weight_history[0:2])\n",
    "print(\"Loss history\", loss_history[0:2])\n",
    "\n",
    "min_index = loss_history.argmin()\n",
    "print(min_index)\n",
    "print(\">>>> min Weight: \", weight_history[min_index])\n",
    "print(\">>>> min Loss: \", loss_history[min_index])\n",
    "\n",
    "# print(we)\n",
    "print(\"---- THe predict loss: \", loss_function(\n",
    "    X_test, y_test, weight_history[min_index]))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Usage\n",
    "\n",
    "Now we could use these data of `min_weight` as the fixed value, into the predict function....\n",
    "\n",
    "The question in tutorial is:\n",
    "- In the future of one week. Use the some money (hypothesis 250$, 50$, 50$), to buy `ads` for my shop\n",
    "- it's the `wechat`, `weibo`, `others`\n",
    "- And how many sales value will be return? by this trained model \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = np.array(DF_ads)  # contain all features\n",
    "X = np.delete(X, [3], axis=1)  # delete the label (row 3)\n",
    "y = np.array(DF_ads.sales)\n",
    "X_train_origin, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=0)\n",
    "\n",
    "\n",
    "X_plan = np.array([250, 50, 50], dtype=float)\n",
    "\n",
    "\"Normalize/Scaler the plan data by the trainning data\"\n",
    "X_train, x_plan = scaler(X_train_origin, X_plan)\n",
    "# X_plan = dummpy(X_plan) # now X_plan is not a 2 dims tensor\n",
    "X_plan = np.append([1], X_plan)\n",
    "\n",
    "print(X_plan.shape)\n",
    "print(X_train.shape)\n",
    "# print(X_plan)\n",
    "# print(X_train[0:2], '\\n', X_plan)\n",
    "\n",
    "\"Use the trained weight\"\n",
    "y_plan = np.dot(weight_history[min_index], X_plan)\n",
    "print(y_plan)\n",
    "\n",
    "\"And it need to reverse scale, to get the true predict value\"\n",
    "y_train_max, y_train_min = y_train.max(), y_train.min()\n",
    "print(y_train_max, y_train_min)\n",
    "y_value = y_plan * y_train_max + y_train_min\n",
    "\n",
    "print(\"===> The finnal sales value that predict by model: \", y_value, \" kilo $\")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "1. Gather,Collect  and Load data\n",
    "2. Preprocesses datatset\n",
    "3. Choose model\n",
    "    - hypothesis $h(x) = wx+b$ or $h(x) = w_0 +w_1x$    ($w_0$ as the bias) \n",
    "    - write as $h(x) = \\theta_0 + \\theta_1 X$ somewhere...\n",
    "4. Loss function:\n",
    "    - and use the gradient descent to minmize the L(w,b) or  $J(\\theta_0, \\theta_1)$\n",
    "$$\n",
    "MSE = L(W,b) = J(\\theta_0, \\theta_1) = \\frac{1}{N} \\sum_{ (X,y)\\in{D}}  (Y-h(X))^2\n",
    "$$\n",
    "\n",
    "\n",
    "5. Adjust the gradient's initial `weight`, `learning rate` (alpha), `iterations`\n",
    "\n",
    "6. Use the test dataset/ use the question X_input to get the output by the weight has been trained by model\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Home works\n",
    "\n",
    "####  练习一\n",
    "\n",
    "**在这一课中，我们花费了一些力气自己从头构造了一个线性回归模型，并没有借助Sklearn库的线性回归函数。这里请大家用Sklearn库的线性回归函数完成同样的任务。怎么做呢？同学们回头看看第1课1.2.3节中的“用Google Colab开发第一个机器学习程序”的加州房价预测问题就会找到答案。**\n",
    "\n",
    "（提示：学完本课内容之后，面对线性回归问题，有两个选择，要么自己构建模型，要么直接调用机器学习函数库里现成的模型，然后用fit方法训练机器，确定参数。）\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 练习二\n",
    "\n",
    "**在Sklearn库中，除了前面介绍过的Linear Regression线性回归算法之外，还有Ridge Regression（岭回归）和Lasso Regression（套索回归）这两种变体。请大家尝试参考Sklearn在线文档，找到这两种线性回归算法的说明文档，并把它们应用于本课的数据集。**\n",
    "\n",
    ">Ridge Regression和Lasso Regression与普通的线性回归在细节上有何不同？下一课中会简单地介绍。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
